Guidelines for Using Shared Libraries

• "Choosing Library Members" explains what routines to include and exclude when you choose library members.

• "Tuning Shared Library Code" covers how to tune shared library code by minimizing global data, improving locality, and aligning for paging.

Choosing Library Members

• Include large, frequently used routines

• Exclude infrequently used routines

• Exclude routines that use much static data

• Make libraries self-contained

Exclude Infrequently Used Routines. Putting these routines in a shared library can degrade performance, particularly on paging systems. Traditional a.out files contain all code they need at run time. By definition, the code in an a.out file is (at least distantly) related to the process. Therefore, if a process calls a function, it may already be in memory because of its proximity to other text in the process.

If the function is in the shared library, a page fault may be more likely to occur, because the surrounding library code may be unrelated to the calling process. Only rarely will any single a.out file use everything in the shared C library. If a shared library has unrelated functions, and unrelated processes make random calls to those functions, the locality of reference may be decreased. The decreased locality may cause more paging activity and, thereby, decrease performance.

Exclude Routines that Use Much Static Data. These modules increase the size of processes. Every process that uses a shared library gets its own private copy of the library's data, regardless of how much of the data is needed.

Library data is static: it isn't shared and can't be loaded selectively with the provision that unreferenced pages may be removed from the working set.

For example, getgrent(3C) is not used by many standard UNIX commands. Some versions of the module define over 1400 bytes of unshared, static data. It probably should not be included in a shared library. You can import global data, if necessary, but not local, static data.

Make Libraries Self-Contained. It's best to make the library self-contained. You can do this by including routines in the shared object. For example, printf(3S) requires much of the standard I/O library. A shared library containing printf(3S), should also contain the rest of the standard I/O routines. This is done with libc.so.1.

If your shared object calls routines from a different shared object, it is best to build in this dependency by naming the needed shared objects on the link line in the usual way. For example:

ld -shared -all mylib.a -o mylib.so -lfoo

This guideline should not take priority over the others in this section. If you exclude some routine that the library itself needs based on a previous guideline, consider leaving the symbol out of the library and importing it.

Tuning Shared Library Code

• Minimize global data

• Organize to Improve locality

• Align for paging

1. Try to use automatic (stack) variables. Don't use permanent storage if automatic variables work. Using automatic variables saves static data space and reduces the number of symbols visible to application processes.

2. Determine whether variables really must be external. Static symbols are not visible outside the library, so they may change addresses between library versions. Only external variables must remain constant.

3. Allocate buffers at run time instead of defining them at compile time. Allocating buffers at run time reduces the size of the library's data region for all processes and, thus, saves memory. Only processes that actually need the buffers get them. It also allows the size of the buffer to change from one release to the next without affecting compatibility. Statically allocated buffers cannot change size without affecting the addresses of other symbols and, perhaps, breaking compatibility.

The cord(1) command rearranges procedures to reduce paging and achieve better instruction cache mapping. You can use cord to see the number of cycles spent in a procedure and the number of times the procedure was executed. The cflow(1) command generates static dependency information. You can combine it with profiling to see what is actually called, as opposed to what may be called.

Align for Paging. The key is to arrange the shared library target's object files so that frequently used functions don't unnecessarily cross page boundaries. When arranging object files within the target library, be sure to keep the text and data files separate. You can reorder text object files without breaking compatibility; the same is not true for object files that define global data.

For example, the IRIX 5.x operating system currently uses 4Kb pages. Using name lists and disassemblies of the shared library target file, the library developers determined where the page boundaries fell.

After grouping related functions, they broke them into page-sized chunks. Although some object files and functions are larger than a single page, most of them are smaller. Then the developers used the infrequently called functions as glue between the chunks. Because the glue between pages is referenced less frequently than the page contents, the probability of a page fault decreased.

After determining the branch table, they rearranged the library's object files without breaking compatibility. The developers put frequently used, unrelated functions together, because they would be called randomly enough to keep the pages in memory. System calls went into another page as a group, and so on. For example, the order of the library's object files became:

Before After

#objects        #objects 
   ...   .          ...
   printf.o         trcmp.o
   fopen.o          malloc.o 
   malloc.o         printf.o
   strcmp.o         fopen.o
   ...  .           ...